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research.  Furthermore,  the  RAE  provides  benchmarks  which  are  used  by  institutions  in 
developing  and  managing  their  research  strategies.  Across  the  UK  as  a  whole,  research 
quality  as  measured  by  the  RAE  has  improved  dramatically  over  the  last  decade 


Table  2:  The  QPIE  quality  assessment  framework 


Quality 

The  intrinsic  excellence  of  the  research  in  world  terms 

People 

The  extent  to  which  the  output  of  trained  staff  meets  the  requirements 
of  employers. 

Impact 

The  potential  for  the  research  output  to  have  a  wider  impact  on  other 
research. 

Explotability 

The  potential  for  the  research  to  contribute  to  wealth  creation  and 
quality  of  life  through  new  or  improved  products,  processes  and 
services. 

In  Australia,  the  Australian  Research  Council  (ARC)  conducted  a  survey  of  researchers 
to  establish  what  researchers  in  each  discipline  regarded  as  appropriate  and 
inappropriate  indicators  for  their  field  [1].  The  survey  found  very  widespread 
agreement  that  publication  of  research  results  in  refereed  journals  is  the  most  relevant 
indicators  of  research  performance.  The  first  seven  most  important  indicators  are  listed 
in  Table  3.  Clearly  these  indicators  are  best  suited  to  publication-oriented  research 
outputs.  Also,  there  are  no  indicators  on  research  outcome. 

Table  3:  Indicators  of  research  quality  in  academic  research  [1] 


1 

Publications  in  refereed  journals 

2 

Peer  reviewed  books 

3 

Keynote  addresses  (at  conferences) 

4 

Conference  proceedings  refereed  papers 

5 

Citation  impact  (publications  weighted  by  journal  citation  impact 

6 

Chapters  in  peer  reviewed  books 

7 

Competitive,  peer  reviewed  grants 

Among  the  Federal  agencies  in  the  United  States,  the  National  Science  Foundation 
(NSF)  has  a  unique  mission:  to  strengthen  the  overall  health  of  U.S.  science  and 
engineering  across  a  broad  and  expanding  frontier.  NSF  invests  in  the  best  ideas  from 
the  most  capable  people,  determined  by  competitive  merit  review.  The  merit  review 
system  [12,13]  is  at  the  very  heart  of  NSF's  selection  of  the  projects  through  which  its 
mission  is  achieved.  NSF  evaluates  proposals  for  research  and  education  projects  using 
two  criteria:  the  intellectual  merit  of  the  proposed  activity  and  the  broader  impacts  of 
the  proposed  activity  on  society.  In  particular,  NSF  uses  a  PIT  framework:  People, 
Ideas  and  Tools: 

PEOPLE  to  develop  a  diverse,  internationally  competitive  and  globally-engaged 
workforce  of  scientists,  engineers  and  well-prepared  citizens.  This  goal  supports  the 
parts  of  NSF’s  mission  that  are  directed  at  (1)  programs  to  strengthen  scientific  and 
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engineering  research  potential;  and  (2)  science  and  engineering  education  programs 
at  all  levels  and  in  all  fields  of  science  and  engineering. 

IDEAS  to  provide  a  deep  and  broad  fundamental  science  and  engineering 
knowledge  base.  This  goal  supports  the  parts  of  NSF’s  mission  directed  at  basic 
scientific  research  and  research  fundamental  to  the  engineering  process. 

TOOLS  to  provide  widely  accessible,  state-of-the-art  science  and  engineering 
infrastructure.  This  goal  supports  the  parts  of  NSF’s  mission  directed  at  (1) 
programs  to  strengthen  scientific  and  engineering  research  potential;  and  (2)  an 
information  base  on  science  and  engineering  appropriate  for  development  of 
national  and  international  policy. 

Therefore  it  is  dear  that  peer  review  is  at  the  heart  of  virtually  all  research  grant 
funding  bodies,  although  there  are  some  differences  in  how  this  peer  review  is 
conducted.  For  instance,  NSF's  merit  review  process  involves  both  mailing  and 
interview,  similar  to  UK's  approach,  while  the  ARC  employs  only  the  mailing  method, 
hence  the  criteria  used  are  less  comprehensive  as  compared  to  that  employed  in  the 
USA  and  the  UK. 

It  is  also  apparent  that  all  the  funding  bodies  employ  multiple  criteria  to  assess  the 
quality  of  research,  confirming  that  the  greater  the  variety  of  measures  used  to  evaluate 
research  quality,  die  greater  is  the  likelihood  of  converging  to  an  accurate  measure  of 
research  quality.  Furthermore,  it  is  interesting  to  note  that  individual  excellence,  and 
increasingly,  team-based  excellence,  have  featured  prominently  as  a  quality  measure. 

3.2  Industrial  research 

The  objectives  of  industrial  R&D  organisations  are  distinctly  different  from  those  of 
universities.  The  quality  of  the  research  undertaken  by  industrial  R&D  organisations  is 
ultimately  judged  via  the  company's  performance  in  terms  of  market  share,  profits, 
sales  and  consumer  satisfaction.  All  these  are  essentially  irrelevant  for  universities.  A 
retrospective  approach  is  required  to  objectively  measure  the  contribution  of  research 
to  company  sales  or  profit.  For  instance,  Unilever's  annual  saving  in  notional  costs  of 
exclusive  licences  and  options  was  estimated  to  be  approximately  equal  to  the  total  cost 
of  Unilever  research  [6],  before  the  value  of  other  outputs  from  research  are  counted. 
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Table  4  summarises  some  of  the  main  indicators  [6]  being  used  in  practice. 
Table  4:  Qualitative  measures  of  quality  in  industrial  R&D 


Quality  Indicators 

Examples 

Professionalism 

Peer  review  &  feedback;  delivery  on  time  and 
budget 

Technology  transfer 

Successful  exploitation 

People 

Skills  and  competence;  mean  age  and  turnover 

Science 

Peer  review;  benchmarking;  latest  facilitates 

Innovations 

Number  of  patents  and  usage 

Business  relevance 

Strategic  focus;  synergy  and  spin-off 

Technology  insurance 

Efficacy  of  responses;  number  of 
incidences  /  surprises 

Knowledge 

management 

Use  of  computer  technology  to  enhance  efficiency 

However,  most  of  the  measures  are  historical,  subjective  and  qualitative.  As  compared 
to  university  research,  the  methodology  for  measuring  the  research  quality  of 
industrial  R&D  organisations  is  far  less  well  developed  and  explored,  a  situation  not 
too  dissimilar  to  that  existing  in  mission-oriented  organisations  as  discussed  in  the  next 
section. 

3.3  Mission-oriented  research 

Nearly  all  mission-oriented  organisations,  such  as  NASA  and  most  of  the  labs  funded 
by  the  Department  of  Defence,  Department  of  Energy  in  the  USA,  and  CSIRO  and 
DSTO  in  Australia,  do  not  have  formal  quality  assessment  schemes.  This  is  to  a  large 
extent  due  to  the  lack  of  requirement  so  far,  as  compared  to  other  research  funding 
bodies,  to  integrate  the  allocation  of  R&D  resources  with  performance  assessment 
mechanisms.  It  is  anticipated  that,  however,  the  pressure  will  intensify  considerably  for 
mission-oriented  organizations  to  introduce  quality  assurance,  as  many  governments 
have  now  legislated  and  mandated  the  integration  of  performance  assessment 
mechanisms  into  the  research  process  to  help  measure  the  effectiveness  of  government 
funded  research  programmes  [14]. 

For  instance,  DSTO  has  introduced  a  quality  review  system  for  its  enabling  research 
programme,  which  accounts  for  about  10%  of  the  total  research  portfolio.  Quality 
reviews  are  conducted  for  every  enabling  R&D  task  through  its  three-year  life  span. 
Normally  the  reviewing  panel  consists  of  three  persons:  two  external  experts  and  one 
internal  expert.  Often  the  diverse  range  of  the  type  of  research  makes  it  difficult  to 
arrive  at  an  absolute  measure  of  quality  across  the  disciplines.  But  it  has  been  noted  by 
one  report  [15]  from  a  recent  review  that  the  research  quality  can  be  assessed  to  some 
degree  by  a  number  of  peer  review  mechanisms,  which  can  be  applied  both  to  the  track 
record  of  the  principal  investigators  and  to  the  project  itself.  A  number  of  quality 
indicators  have  been  suggested,  as  listed  in  Table  5. 
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Table  5:  Quality  indicators  for  strategic  research  in  DSTO 


FT 

Refereed  iournal  and  conference  publications. 

_2j 

Adoption  of  research  outcomes  by  clients. 

3 

National  and  international  enquires  for  help  in  problem  solving. 

4 

Requests  from  other  researchers  to  undertake  sabbatical  or  secondment  working 
with  the  researchers. 

V 

Reauests  to  license  intellectual  property  arising  from  a  research  project. 

6 

Benchmarking  with  competing  research  groups  on  an  international  basis. 

7 

Other  evidence  of  peer  esteem  for  the  work  being  done. 

4.  A  Proposed  Quality  Assessment  System  for 
Mission-Oriented  Research 

4.1  Common  quality  measures 

Impact/Outcome 

The  impact  or  outcome  of  research  is  arguably  the  most  important  indication  of 
research  quality,  especially  for  applied  research  or  experimental  development.  In  the 
case  of  mission-oriented  R&D  organisations,  this  means  adoption  of  research  outcomes 
by  customers  who  sponsor  the  research.  This  is  certainly  the  most  important  indicator 
for  industrial  R&D  organisations.  Other  quality  indicators  include  spin-off  and  patents, 
which  are  generally  most  relevant  only  to  industrial  R&D  organisations. 

By  its  nature,  strategic  research  as  conducted  by  universities  and  mission-oriented 
organisations  has  a  long  lead-time  to  yield  a  useful  outcome,  and  hence  is  unlikely  to 
result  in  an  immediate  outcome  within  the  duration  of  the  project.  Therefore  the 
impact  of  strategic  research  cannot  be  measured  instantly  and  a  retrospective  approach 
is  required.  In  this  case,  the  track  record  of  the  researchers  is  the  most  appropriate 
retrospective  quality  indicators:  past  success  is  the  best  predictor  of  future 
performance. 
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Executive  Summary 


This  report  is  based  on  an  essay  written  during  the  course  of  Graduate  Certificate  in 
Scientific  Leadership,  Melbourne  University,  Australia. 

The  quality  of  research  is  vital  to  any  research  and  development  (R&D)  organisation, 
including  academic,  industrial,  and  government-funded  mission-oriented 
organisations.  With  the  increasing  pressures  on  public  funding  for  research  and  from 
within  the  research  enterprise  itself,  it  is  becoming  increasingly  important  to  ensure 
quality  control,  to  demonstrate  efficiency  in  the  use  of  resources,  and  to  provide 
accountability.  The  key  issue  that  needs  to  be  addressed  can  be  separated  into  two 
major  questions:  (a)  how  should  the  quality  of  research  be  defined,  and  (b)  how  should 
this  be  measured.  This  issue  is  critical  to  planning  and  priority  setting  in  scientific  and 
engineering  research,  and  decision  making  in  allocation  of  research  funding. 

This  report  explores  suitable  measures  to  assess  the  quality  of  mission-oriented 
research,  with  the  view  of  identifying  appropriate  strategies  to  improve  research 
performance.  A  peer-reviewed-objective-dependent  (PROD)  evaluation  system  is 
proposed  to  cater  for  the  broad  spectrum  of  research  undertaken  by  mission-oriented 
research  organisations,  ranging  from  strategic  research  through  to  application  and 
development.  In  the  proposed  PROD  system,  quality  indicators  and  the  pertinent 
weightings  are  adjusted  to  be  consistent  with  the  research  objectives.  It  is  possible  to 
obtain  a  single  number  to  quantify  research  quality  using  the  bibliometrics  method,  in 
which  the  ratings  are  determined  through  peer  reviewing. 

This  quality  assessment  system  could  provide  a  useful  tool  for  planning,  priority 
setting,  and  quality  assurance  in  research  and  development. 


Authors 


Chun  H.  Wang 

Airframes  and  Engines  Division 

After  obtaining  his  Ph.D  from  the  University  of  Sheffield ',  UK,  in 
1990,  Dr.  Chun  Wang  worked  as  a  research  fellow  at  the 
University  of  Sheffield  (1990-93)  and  Sydney  University  (1993- 
1994)  on  fatigue,  reliability  assessment  and  biomimatics.  Between 
1994  and  1995  he  worked  as  a  Lecturer  at  Deakin  University  and 
undertook  research  in  mechatronics  and  advanced  composite 
materials.  Since  joining  DSTO  in  1995  he  has  been  working 
mainly  in  the  areas  of  fatigue  crack  growth  modelling,  fatigue  life 
prediction,  computational  mechanics,  and  composite  repairs.  Dr. 
Wang  is  currently  a  Principal  Research  Scientist  and  the 
Functional  Head  of  Damage  Mechancis  in  the  Airframes  and 
Engines  Division. 


Contents 

1.  INTRODUCTION . 1 

2.  CLASSIFICATION  OF  R&D  AND  RESEARCH  OBJECTIVES . 3 

3.  REVIEW  OF  QUALITY  MEASURES . 6 

3.1  Academic  research . 6 

3.2  Industrial  research . 8 

3.3  Mission-oriented  research . 9 

4.  A  PROPOSED  QUALITY  ASSESSMENT  SYSTEM  FOR  MISSION-ORIENTED 
RESEARCH  10 

4.1  Common  quality  measures . 10 

4.2  A  Peer-Reviewed-Objective-Dependent  (PROD)  system . 11 

5.  CONCLUSION . 13 

6.  ACKNOWLEDGEMENTS . 13 

7.  REFERENCES . 14 


DSTO-GD-0276 


1.  Introduction 

The  quality  of  research  is  vital  to  any  research  and  development  (R&D)  organisation, 
including  academic,  industrial,  and  government-funded  mission-oriented 
organisations.  With  the  increasing  pressures  on  public  funding  for  research  and  from 
within  the  research  enterprise  itself,  it  is  becoming  increasingly  important  to  ensure 
quality  control,  to  demonstrate  efficiency  in  the  use  of  resources,  and  to  provide 
accountability.  The  key  issue  that  needs  to  be  addressed  can  be  separated  into  two 
major  questions:  (a)  how  should  the  quality  of  research  be  defined,  and  (b)  how  should 
this  be  measured.  This  issue  is  critical  to  planning  and  priority  setting  in  scientific  and 
engineering  research,  and  decision  making  in  allocation  of  research  funding. 
Consequently  extensive  studies  have  been  carried  out  to  identify  suitable  indicators  or 
criteria  for  assessing  the  quality  of  research,  particularly  by  government  funding 
agencies  for  university  research  [1-5]. 

Virtually  all  the  major  research  funding  bodies  worldwide  now  use  composite 
indicators  to  asses  various  aspects  of  quality:  (i)  research  proposal,  (ii)  track  record  of 
researchers,  (iii)  intrinsic  merit  of  the  research  and  (iv)  relevance  of  the  research.  This 
has  been  brought  about  by  government  and  the  public  trying  to  define  more  precisely 
the  purposes  for  which  they  invest  in  scientific  research  and  to  assess  the  results  of  that 
investment.  It  is  often  argued  that  investing  in  second-hand,  second  quality,  research  is 
a  waste  of  money.  Therefore  both  the  selection  and  continuation  of  research 
programmes  must  be  made  on  the  basis  of  outstanding  science  and  potential 
contribution  to  the  organisation's  mission.  This  requires  the  establishment  of 
performance  indicators  to  be  used  in  measuring  the  research  quality,  goal  relevance, 
service  levels,  and  outcomes  of  each  program  activity. 

Parallel  pressure  also  exists  for  industrial  research  organisations  to  link  research 
programmes  more  closely  with  strategic  corporate  goals  and  to  increase  research 
performance  and  productivity,  due  to  the  increasing  world  competition  and  the  trends 
toward  downsizing.  Most  companies  that  invest  in  R&D  have  shown  intense  interest  in 
measuring  research  performance  and  effectiveness  [6].  Therefore,  it  has  become  a  norm 
that  value-for-money  has  to  be  demonstrated  both  in  terms  of  efficiency  and  of  impact, 
which  can  be  regarded  as  indicators  of  quality. 

Excellence  is  often  explicitly  stated  in  the  corporate  goals  of  mission-oriented  R&D 
organisations.  For  example,  the  following  is  an  extract  from  the  purpose  statement  of 
the  Defence  Science  and  Technology  Organisation  (DSTO),  Australia: 

“DSTO’s  business  is  to  enhance  the  effectiveness  of  the  Australian  Defence 
Organisation  (ADO)  through  the  application  of  science  and  technology.  We 
have  a  particular  responsibility  to  support  the  ADO  in  those  capabilities  and 
technologies  in  which  excellence  would  be  most  relevant  to  the  direct 
defence  of  Australia  and  the  policy  of  self-reliance.”. 
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The  National  Aeronautics  and  Space  Administration  (NASA),  USA,  places  an  even 
stronger  emphasis  on  "excellence"  and  "quality",  as  reflected  in  its  mission  statement: 

"We  are  committed  to  demonstrating  and  promoting  excellence  and 
continually  improving  processes,  products,  and  services  to  better  satisfy  our 
customers'  needs  and  requirements.  We  utilize  quality-focused  leadership 
and  management,  as  well  as  scientific,  engineering,  and  technical 
excellence  to  provide  our  customers  with  highly  valued  products  and 
services  in  the  most  cost-effective,  timely,  and  safe  manner." 

However,  by  its  very  nature  R&D  is  not  deterministic  and  is  therefore  difficult  to 
quantify  and  assess  in  a  numerical  manner.  The  definition  of  "quality"  itself  is 
contextual  and  dependent  on  one's  perspective.  For  example,  this  is  dependent  on  the 
organisation  and  would  differ  between  a  large  multi-national  company,  a  small 
national  one,  and  academia  [6].  It  is  noted  that  only  a  few  aspects  of  R&D  can  be 
quantified  (e.g.,  cost),  and  many  aspects,  such  as  outputs  and  outcomes,  are  hard  to 
quantify.  Therefore  measuring  the  quality  of  research  and  development  is  fraught  with 
difficulty.  Nevertheless,  it  is  very  important  to  assess  the  research  performance  for  a 
number  of  reasons: 

(i)  To  provide  meaningful  information  for  external  stakeholders,  including  the 
academic  community; 

(ii)  To  be  accountable  to  the  government  and  public  who  invest  in  the  research; 

(iii)  To  foster,  maintain  and  improve  the  technical  quality  of  the  research 
programme;  and 

(iv)  To  provide  a  useful  input  to  business  planning  processes  by  identifying  and 
redirecting  or  terminating  wayward  research. 

In  the  case  of  DSTO,  whose  main  output  is  scientific  and  engineering  advice,  the 
credibility  of  DSTO's  advice  is  strongly  dependent  on  its  quality  or  excellence  of  its 
research  program. 

To  objectively  assess  the  quality  of  research,  it  is  essential  that  any  indicators  or 
measures  should  be  transparent,  objective,  and  collectable,  which  will  be  the  emphasis 
of  this  report.  In  the  present  context,  research  quality  is  defined  as  the  degree  of 
excellence  in  the  technical  work  itself  (output)  and  exploitation  or  applications 
(outcome).  In  other  words,  the  quality  of  any  R&D  needs  to  be  measured  and 
interpreted  in  terms  of  the  value  of  research,  both  internal  and  external  [3].  Internal 
criteria  arise  from  within  the  science  itself,  and  are  basically  criteria  of  efficiency  or 
productivity —how  well  the  research  has  been  or  will  be  conducted.  Internal  criteria  are 
necessary  criteria  for  the  support  of  any  R&D.  External  criteria  are  criteria  of  utility— 
that  is,  they  measure  the  degree  to  which  the  given  research  is,  in  the  broadest  sense, 
useful  outside  the  field  itself.  In  the  case  of  mission-oriented  organisations,  this  means 
that  the  research  should  lead  to  ultimate  applications.  Research  quality  should  also  be 
judged  against  the  motivations  or  objectives  of  the  R&D  undertaken,  which  may  differ 
significantly  between  universities,  industrial  R&D  organisations,  and  mission-oriented 
R&D  organisations  (such  as  DSTO).  While  the  measures  of  internal  quality  may  be 
common  across  these  different  types  of  research,  indicators  measuring  the  external 
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quality  would  be  strongly  dependent  on  the  objectives  of  the  research  programme. 
Therefore  a  set  of  objective-dependent  indicators  are  required  for  different  type  of 
research. 

The  present  report  will  focus  on  the  indicators  applicable  to  mission-oriented  research, 
while  brief  discussion  of  the  various  indicators  employed  for  academic  and  industrial 
research  will  be  presented  for  comparison  purposes.  Section  2  discusses  the  objectives 
of  different  types  of  research  being  carried  out  in  universities,  mission-oriented  and 
industrial  research  organisations.  A  review  and  critical  assessment  of  the  quality 
indicators  and  the  measuring  process  that  are  currently  employed  by  various  research 
funding  bodies  is  presented  in  Section  3,  together  with  the  methodologies 
underpinning  these  current  practices.  A  peer-reviewed-objective-dependent  (PROD) 
approach  is  proposed  in  Section  4  for  mission-oriented  R&D  organisations  with  a 
strong  focus  on  meeting  customer  needs. 


2.  Classification  of  R&D  and  Research  Objectives 


Broadly  speaking  there  are  three  main  performers  of  R&D:  (1)  universities;  (2) 
industrial  R&D  organisations;  and  (3)  government  laboratories.  Examples  of  the  last 
group  of  performers  include  laboratories  funded  by  governments  and  the  Department 
of  Defence  in  many  countries  around  the  world,  which  will  be  denoted  in  this  report  as 
mission-oriented  R&D  organisations.  A  good  classification  framework  of  the  broad 
spectrum  of  scientific  and  engineering  research  is  provided  by  the  OECD  [7],  which 
sets  out  the  internationally  agreed  categories  for  surveys  of  R&D:  basic  research, 
applied  research  and  experimental  development.  This  classification  was  later  refined  to 
sub-divide  the  basic  research  into  pure  research  and  strategic  research  [8].  This 
refinement  is  particularly  important  as  the  strategic  research  covers  the  enabling 
research  undertaken  by  most  government  laboratories  and  large  science-based 
companies  (in  which  it  typically  accounts  for  about  10  per  cent  of  the  R&D  effort).  The 
specific  objectives  of  these  different  types  of  research  are  summarised  in  Table  1. 


Table  1:  Classification  framework  and  objectives  of  R&D  activities 


Type  of  R&D 

Objectives 

Basic  Research 

Pure  research 

Advancement  of  knowledge,  no  efforts  on 
application  or  transition  of  results. 

Strategic  research 

Broadening  knowledge  base  and  basis  for 
solution  of  current  or  future  practical 
problems. 

Applied  Research 

Determining  possible  uses  for  findings  of 
strategic  research  and  solving  already 
recognised  problems. 

Application  and  Development 

Producing  new  or  improved  materials, 
products,  devices,  prototypes,  and  systems. 
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To  a  very  large  extent,  the  quality  of  research,  i.e.,  its  impact  and  significance,  is  a 
reflection  of  the  degree  of  innovation.  Two  very  appealing  models  that  have  dominated 
the  discussion  of  innovation  over  the  past  three  decades  are  the  "science-push"  model 
and  the  "demand-pull"  model,  as  depicted  as  follows  [8]: 


Curiosity-oriented  Applied  Experimental  ^  Innovation 

research  *  research  “  development 


Figure  1:  Science-push  model  of  innovation 


Market  ^  Applied  Experimental  Innovation 

demand  ®  research  ™  development 


Figure  2:  Demand-pull  model  of  innovation 


Considerable  effort  has  been  devoted  in  the  past  to  ascertain  which  of  the  two 
competing  models  most  accurately  characterises  the  innovation  process,  trying  to 
obtain  systematic  statistical  evidence  of  most  important  factors  contributing  to 
innovation  [8].  The  results  of  these  studies  indicated  that  most  innovations  in  practice 
fit  neither  of  these  simple  models.  In  other  words,  the  innovation  process  cannot 
generally  be  represented  as  a  linear  sequence  of  events  with  a  single  cause.  Instead,  it 
has  been  found  that  both  "recognition  of  a  technical  opportunity"  (science-push)  and 
"recognition  of  a  need"  (demand-pull)  have  been  identified  as  key  factors  of 
approximately  equal  importance  in  the  most  decisive  events  led  to  innovation. 

The  complicated  interaction  between  science-push  and  demand-pull  is  prominent  in 
the  Defence  Science  and  Technology  Organisation  (DSTO),  which  provides  advice  on 
the  application  of  science  and  technology  best  suited  to  Australia's  defence  and 
security  needs.  DSTO  conducts  R&D  in  a  wide  variety  of  defence-relevant  fields, 
focusing  on  areas  which  are  unique  to  Australia's  needs  or  otherwise  central  to 
national  self-reliance.  As  emphasised  in  the  Defence  White  Paper,  science  and 
technology  play  a  central  role  in  the  success  of  Australia's  defence.  Australia  has  a 
relatively  small  population,  and  therefore  Australia's  defence  policy  gives  priority  to 
capabilities  that  rely  on  high  technology  rather  than  on  large  numbers  of  personnel. 

Australia  has  four  broad  objectives  for  its  defence  science  and  technology  advice.  They 
are  (i)  to  position  Australia  to  exploit  future  developments  in  technology  which  show 
promise  for  defence  applications;  (ii)  to  ensure  that  Australia  is  an  informed  buyer  of 
equipment;  (iii)  to  develop  new  defence  capabilities  as  required;  and  (iv)  to  support 
existing  capabilities  by  increasing  operational  performance  and  reducing  the  costs  of 
ownership,  including  life-extension  programs. 
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During  1992-93,  DSTO  introduced  several  Program  Improvement  initiatives  to  provide 
extensive  visibility  of  the  Science  and  Technology  program  and  to  involve  the  customer 
group  more  extensively  in  program  review  and  priority  setting.  The  Program 
Improvement  process  has  led  directly  to  a  substantial  increase  in  customer  satisfaction 
with  DSTO's  output  and  the  balance  of  its  products  and  services.  However, 
considerable  pressure  within  each  research  area,  together  with  increasing  customer 
awareness  and  requirements,  has  led  to  a  sharpening  of  the  competition  for  resources. 
A  consequence  of  this  process  has  been  to  reduce  the  quantity  of  long-term  enabling 
R&D.  It  is  nevertheless  recognised  that  DSTO  must  manage  and  sustain  a  credible 
program  of  enabling  R&D  independently  of  immediate  customer  requirements  but 
clearly  mindful  of  future  applications.  The  current  structure  of  DSTO's  research 
program  can  be  approximately  grouped  into  three  categories  [9]:  enabling  research, 
applied  research  and  development,  as  depicted  in  Fig.3.  Enabling  R&D  may  have  no 
immediate  practical  outcome  but  it  should  have  the  long-term  potential  for  application 
and  exploitation.  Enabling  R&D  may  provide  a  stream  of  novel  and  innovative  ideas  in 
areas  of  high  technological  risk  and  potentially  high  payoff. 

By  contrast  to  industrial  organisations  whose  main  outputs  are  products-for-profit, 
DSTO's  main  output  is  scientific  and  engineering  advice.  The  credibility  of  DSTO's 
advice  is  therefore  strongly  dependent  on  its  quality  or  excellence  of  its  research 
program.  The  quality  of  its  research  also  serves  as  a  good  indicator  whether  DSTO 
follows  the  best  practice  in  R&D  management.  However,  due  to  the  different  nature  of 
DSTO's  research  as  compared  to  that  of  university  and  industrial  R&D  organisations, 
conventional  measures  or  indicators  that  have  been  widely  used  to  evaluate  the 
university  research  may  be  applicable  only  to  the  enabling  research  of  DSTO.  Other 
quality  measures  are  required  for  the  other  two  categories  of  research:  applied  research 
and  development.  This  will  be  the  focus  of  Section  4. 


Figure  3:  Push-pull  model  of  DSTO's  research  and  development. 
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To  identify  suitable  indicators  that  may  be  applicable  to  mission-oriented  R&D 
organisations,  it  is  useful  to  review  the  quality  indicators  employed  by  the  major 
research  funding  organisations. 


3.  Review  of  Quality  Measures 


The  three  main  methods  of  measuring  research  quality  are  retrospective,  qualitative 
and  quantitative.  For  the  three  main  R&D  performers  discussed  in  Section  2, 
universities,  industrial  organisations  and  mission-oriented  government  organisations, 
different  sets  of  quality  measure  are  required  for  each  due  to  the  differences  in  the 
objective  of  R&D  undertaken  by  each  organisation.  The  following  review  will  focus  on 
existing  indicators  that  have  been  developed  for  academic  research,  industrial  research, 
and  mission-oriented  research. 

3.1  Academic  research 

The  existing  quality  assessment  as  employed  in  decision  making  of  research  funding 
bodies  to  assist  the  allocation  of  research  grants  is  centred  around  the  peer  review 
method,  which  represents  evaluation  by  experts  in  the  field.  This  is  the  method  of 
choice  in  practice  in  many  countries,  including  the  USA,  UK,  and  Australia. 

For  instance,  the  Higher  Education  Funding  Councils  in  the  UK  conduct  a  Research 
Assessment  Exercise  (RAE),  aimed  to  enable  the  higher  education  funding  bodies  to 
distribute  public  funds  for  research  selectively  on  the  basis  of  quality  [10,11]. 
Institutions  deemed  to  be  conducting  the  best  research  receive  a  larger  proportion  of 
the  available  grant  so  that  the  infrastructure  for  the  top  level  of  research  in  the  UK  is 
protected  and  developed.  Typically  the  quality  assessment  is  carried  out  through  the 
peer  reviewing  process,  coupled  with  interviewing  and  site  visits  by  a  panel  of  experts. 
It  takes  place  every  four  to  five  years.  The  RAE  provides  quality  ratings  for  research 
across  all  disciplines.  Panels  use  a  standard  scale  to  award  a  rating  for  each  submission. 
Ratings  range  from  1  to  5,  according  to  how  much  of  the  work  is  judged  to  reach 
national  or  international  levels  of  excellence.  The  assessment  involves  a  panel  of 
internationally  recognised  experts  to  benchmark  the  quality  on  an  international  scale. 
The  report  provided  by  the  panel  amounts  to  a  snapshot  view  of  people  well  equipped 
to  recognise  excellence.  Typically  the  evaluation  framework  encompasses  Quality, 
People,  Impact  and  Exploitability  (QPIE),  as  summarised  in  Table  2. 

It  is  important  to  note  that  the  results  of  the  research  assessment  have  a  much  wider 
value  than  its  immediate  purpose.  For  example,  it  can  be  helpful  in  guiding  funding 
decisions  in  industry  and  commerce,  charities  and  other  organisations  that  sponsor 
research.  It  also  gives  an  indication  of  the  relative  quality  and  standing  of  UK  academic 
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Publications 

One  important  common  indicator  of  research  quality  among  university,  industrial,  and 
mission-oriented  R&D  organisations  is  refereed  publications.  There  are  a  number  of 
very  important  reasons  for  this  strong  emphasis  on  publications: 

1.  For  enabling  research,  the  technical  issues  being  addressed  have  a  generic 
relevance  within  the  appropriate  discipline,  so  that  any  significant  progress 
or  achievement  should  be  publishable. 

2.  The  more  rigorous  refereeing  process  involved  in  publishing  journal  papers 
serves  to  avoid  duplication  or  "re-inventing  the  wheel"  type  of  research. 

3.  Acceptance  via  peer-review  process  represents  a  highly  objective  peer- 
recognition  of  research  quality. 

Therefore  the  number  of  publications  is  not  merely  a  measure  of  productivity,  but  also 
a  very  important  indication  of  quality.  It  should  be  noted,  however,  that  the  calibre  and 
peer  regard  for  journals  varies  widely.  The  citation  impact  is  therefore  used  as  an 
indicator  of  relative  quality.  Furthermore,  a  citation  analysis  may  be  required  to 
determine  peers'  assessment  of  the  impact  of  the  cited  publication.  Also,  in  a  mission- 
focused  context,  journal  publications  should  not  be  over-emphasised,  as  that  may 
adversely  modify  behaviour  by  directing  research  away  from  the  most  relevant 
engineering  problems  with  applications  to  industry  towards  projects  which  lead  more 
readily  to  journal  papers. 

People 

The  skills  and  competence  of  the  researchers  are  also  a  very  important  quality 
indicator.  There  is  no  doubt  that  the  skills  and  competence  of  the  people  are  essential  to 
achieve  international  best  practice  in  R&D.  Therefore  it  is  important  to  recruit,  train 
and  retain  people  with  the  best  performance  track  record,  and  let  them  lead  the 
research  and  development. 


4.2  A  Peer-Reviewed-Objective-Dependent  (PROD)  system 

From  the  above  analysis  two  things  become  clear.  First  of  all,  quality  assessment  needs 
to  be  based  on  peer  review.  In  the  case  of  research  publications,  peer  view  is  by 
international  experts  or  peers  engaging  in  the  same  field  of  research.  For  application 
and  development  where  the  products  are  mostly  in  the  form  of  scientific  advice  given 
to  the  customers,  an  assessment  of  quality  requires  direct  inputs  as  feedback  by  the 
customers.  Secondly,  quality  is  value-system  dependent  and  has  to  be  assessed  against 
the  objectives  of  the  research.  This  implies  that  a  certain  quality  measure,  like 
publication,  may  be  more  important  for  strategic  research  but  less  relevant  for 
experimental  development  or  commercial  R&D.  Therefore  the  weighting  of  quality 
measures  need  to  be  adjusted  in  accordance  with  the  research  objective.  In  other 
words,  an  appropriate  quality  assessment  scheme  for  mission-oriented  R&D 
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organisation  has  to  be  objective-dependent.  Most  mission-oriented  R&D  organisations 
conduct  a  mixture  of  R&D,  including  strategic  research,  applied  research,  and 
experimental  development.  For  instance,  DSTO  conducts  about  10%  enabling  research, 
30%  applied  research  and  60%  application  and  development,  as  depicted  in  Fig.3. 

The  proposed  new  system,  as  illustrated  in  Table  6,  aims  to  cater  for  the  diverse 
spectrum  of  R&D  undertaken  by  mission-oriented  organisations.  Firstly,  three  different 
sets  of  quality  indicators  are  required  for  the  three  different  levels  of  research. 
Secondly,  the  weightings  of  quality  indicators  would  need  to  be  adjusted  to  fit  the 
characteristics  of  the  field.  For  instance,  in  the  case  of  strategic  research  aiming  to 
broaden  the  science  and  technology  base,  the  major  quality  measures  should  be  similar 
to  those  widely  employed  by  leading  research  grant  funding  bodies  to  measure  the 
research  quality  of  university  research.  In  the  case  of  development,  it  is  more 
appropriate  to  place  higher  emphasis  on  outcome  and  customer  satisfaction,  and 
publications  would  be  less  critical  although  still  important.  It  should  be  noted  that 
although  any  one  indicator  does  not  guarantee  quality,  the  lack  of  these  indicators 
would  certainly  mean  the  lack  of  quality  or  excellence.  Furthermore,  a  cardinal  rule  for 
the  proper  use  of  quantitative  indicators  is  broken  when  any  single  indicator  is  relied 
upon. 

To  achieve  a  truly  quantitative  quality  assessment,  a  bibliometrics  approach  has  to  be 
adopted:  each  indicator  is  ranked  (e.g.,  using  a  rating  between  1  and  10),  the  rating  is 
multiplied  by  the  weighting,  and  the  results  are  added  to  arrive  at  the  figure  of  merit. 
This  will  provide  an  objective  method  to  benchmark  one  organisation's  R&D 
performance  against  that  of  other  similar  institutions.  In  the  case  of  DSTO,  this  would 
help  to  achieve  efficiency  and  effectiveness  to  ensure  that  DSTO  is  providing  "best 
value  for  service". 

It  should  be  cautioned  here  that  it  is  important  to  ensure  that  the  indicators  do  form  an 
orthogonal  set  as  much  as  possible,  so  there  will  be  no  multiple  counting  to  skew  the 
results.  In  addition,  the  rating  of  each  quality  indicator,  such  as  research  publication,  is 
not  simply  a  number  counting.  The  marginal  utility  theory  would  suggest  that  that 
while  it  might  be  twice  as  valuable  for  researcher  to  publish  two  papers  per  year 
compared  to  one  paper,  it  would  probably  not  be  twice  as  valuable  if  the  researcher 
were  to  publish  100  papers  per  year  as  opposed  to  50.  Further  research  is  required  to 
identify  the  utility  functions  for  these  indicators. 
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Table  6:  Research  objectives,  quality  indicators  and  weighting 


Research  Objective 

Quality  indicator 

Weightin 

£ 

Strategic  research: 

Broaden  science  and  technology 
base  for  solution  of  current  and 
future  practical  problems. 

1.  Refereed  publications. 

40% 

2.  Transition  of  research  results  to 
applied  research  or 
development. 

40% 

3.  Technical  leadership:  citations 
and  track  record  of  researchers. 

10% 

4.  Peer  recognition:  invited 
lecture  and  keynote  talk  at 
conferences. 

10% 

Applied  research: 

Solve  problems  using  established 
methodologies  and  principles, 
and 

1.  Conversion  of  science  to 
technology. 

50% 

2.  Publications  (external  and 
internal). 

30% 

3.  Professionalism:  track  record  of 
researchers 

10% 

4.  Peer  review  and  benchmarking 
to  ensure  international  best 
practice. 

10% 

Application  and  development: 
Provide  ready-to-implement 
solutions  to  clients,  and  produce 
new  or  improved  materials, 
products  and  systems. 

1.  Outcome:  acceptance  of  advice 
by  the  clients. 

50% 

2.  Professionalism:  customer 
satisfaction  and  best  practice. 

20% 

3.  Publications  (external  and 
internal) 

20% 

4.  Spin-off,  synergy,  patents 

10% 

5.  Conclusion 

A  critical  review  and  comparison  has  been  conducted  on  quality  measures  currently 
being  employed  to  measure  the  performance  of  university  research,  industrial  research, 
and  mission-oriented  research.  A  peer-reviewed-objective-dependent  (PROD) 
evaluation  system  is  proposed  for  mission-oriented  research.  This  new  quality 
assessment  system  advocates  the  use  of  different  quality  indicators  that  are  consistent 
with  the  research  objectives. 
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